Systems and methods for image-based object modeling using multiple image acquisitions or reconstructions

ABSTRACT

Systems and methods are disclosed for integrating imaging data from multiple sources to create a single, accurate model of a patient&#39;s anatomy. One method includes receiving a representation of a target object for modeling; determining one or more first anatomical parameters of the target anatomical object from at least one of one or more first images of the target anatomical object; determining one or more second anatomical parameters of the target anatomical object from at least one of one or more second images of the target anatomical object; updating the one or more first anatomical parameters based at least on the one or more second anatomical parameters; and generating a model of the target anatomical object based on the updated first anatomical parameters.

RELATED APPLICATION

This application is a continuation of and claims the benefit of priorityfrom U.S. patent application Ser. No. 14/254,491, filed Apr. 16, 2014,which is incorporated by reference herein in its entirety.

FIELD OF THE INVENTION

Various embodiments of the present disclosure relate generally tomedical imaging and related methods. More specifically, particularembodiments of the present disclosure relate to systems and methods forimage-based object modeling using multiple image acquisitions orreconstructions.

BACKGROUND

Medical imaging and extraction of anatomy from imaging is important, asevidenced by the many means of medical imaging available. Common formsof medical imaging include computed tomography (CT) scans, magneticresonance imaging, intravascular ultrasound, intravascular opticalcoherence tomography, angiography, and histopathology optical images. CTscans are x-ray images of “slices” of a scanned object. For example, CTscans are commonly images taken as cross-sectional slices, perpendicularto the long axis of the body. Cardiac CT scans may include calcium-scorescreening and/or angiography. Calcium score screening scans may be usedto detect calcium deposits in coronary arteries, contributing topredictions of heart problems. CT angiography is CT scanning includingintravenous (IV) contrast dye to better show blood vessels and organs.Although also capable of producing tomographic images, magneticresonance (MR) imaging uses magnetic field properties to create theimages. Because CT and MRI images are produced differently, resultantimages highlight different tissue properties. MR images offer betterquality in soft tissue images than CT scans; CT scans image bone andblood vessels in addition to soft tissue, although the soft tissuedetail is inferior to that of MR images. Depending on the anatomy ofinterest and purpose of imaging, CT and MR may be consideredcomplimentary imaging techniques.

Intravascular ultrasound (IVUS) is a type of imaging that visualizes theinside of blood vessels. Whereas CT and MR methods involve images takenas slices of a patient body, IVUS images are achieved via a cathetertraveling through an artery or vein. Thus, IVUS images may essentiallyshow cross-sections of the artery or vein, from the center of a bloodvessel, out through the vessel wall and whatever diseased portion mayexist at the wall. Intravascular optical coherence tomography (OCT) isan optical analog of the ultrasound imaging of IVUS. IVUS and OCT areanalogous imaging modalities, but OCT's use of light (in place of sound)offers higher resolution images than IVUS. Briefly discussed in thecontext of CT scans, angiography is an imaging technique that employs aninjection of a contrast agent into the blood stream to better showvessels or vessel openings. While CT angiography may be preferable forcoronary disease detection, MR angiography is a viable alternative.Histopathological optical imaging includes visualization of tissue on amicroscopic level. Histopathological imaging can be used to identifytissue or detect for various biomarkers. One common prerequisite for theanalysis of histopathological images is the localization of cells,tissue or other anatomical and cellular objects within the images.

Based on images from techniques described above, anatomical models maybe extracted to measure one or more properties of a patient's anatomy(e.g., a tumor or cardiac volume) or to support biophysical simulation(e.g., fluid simulation, biomechanical simulation, electrophysiologicalsimulation, etc.). In order to accurately measure anatomical propertiesor predict physiological phenomena via simulation, a very precisepatient-specific model must be created of the target anatomy. Imagingand subsequent extraction of anatomical models of the heart, forexample, is of special importance. For instance, such imaging andmodeling may provide evaluation of coronary artery disease, such as whena patient is suffering from chest pain, and/or a more severemanifestation of disease, such as myocardial infarction, or heartattack.

Patients suffering from chest pain and/or exhibiting symptoms ofcoronary artery disease may be subjected to one or more tests that mayprovide some indirect evidence relating to coronary lesions. Forexample, noninvasive tests may include electrocardiograms, biomarkerevaluation from blood tests, treadmill tests, echocardiography, singlepositron emission computed tomography (SPECT), and positron emissiontomography (PET). These noninvasive tests, however, typically do notprovide a direct assessment of coronary lesions or assess blood flowrates. The noninvasive tests may provide indirect evidence of coronarylesions by looking for changes in electrical activity of the heart(e.g., using electrocardiography (ECG)), motion of the myocardium (e.g.,using stress echocardiography), perfusion of the myocardium (e.g., usingPET or SPECT), or metabolic changes (e.g., using biomarkers). Forexample, anatomic data may be obtained noninvasively using coronarycomputed tomographic angiography (CCTA). CCTA may be used for imaging ofpatients with chest pain and involves using CT technology to image theheart and the coronary arteries following an intravenous infusion of acontrast agent.

However, single images may be insufficient to create ideal models. Theforegoing general description and the following detailed description areexemplary and explanatory only and are not restrictive of thedisclosure.

SUMMARY

According to certain aspects of the present disclosure, systems andmethods are disclosed for modeling at least a portion of a patient'sanatomy. One method includes: receiving a representation of a targetobject for modeling; determining one or more first anatomical parametersof the target anatomical object from at least one of one or more firstimages of the target anatomical object; determining one or more secondanatomical parameters of the target anatomical object from at least oneof one or more second images of the target anatomical object; updatingthe one or more first anatomical parameters based at least on the one ormore second anatomical parameters; and generating a model of the targetanatomical object based on the updated first anatomical parameters.

In accordance with another embodiment, a system for modeling at least aportion of a patient's anatomy, comprises: a data storage device storinginstructions for modeling based on patient-specific anatomic image data;and a processor configured to execute the instructions to perform amethod including receiving a representation of a target object formodeling; determining one or more first anatomical parameters of thetarget anatomical object from at least one of one or more first imagesof the target anatomical object; determining one or more secondanatomical parameters of the target anatomical object from at least oneof one or more second images of the target anatomical object; updatingthe one or more first anatomical parameters based at least on the one ormore second anatomical parameters; and generating a model of the targetanatomical object based on the updated first anatomical parameters.

In accordance with yet another embodiment, a non-transitory computerreadable medium for use on a computer system containingcomputer-executable programming instructions for modeling at least aportion of a patient's anatomy is provided. The method includes:receiving a representation of a target object for modeling; determiningone or more first anatomical parameters of the target anatomical objectfrom at least one of one or more first images of the target anatomicalobject; determining one or more second anatomical parameters of thetarget anatomical object from at least one of one or more second imagesof the target anatomical object; updating the one or more firstanatomical parameters based at least on the one or more secondanatomical parameters; and generating a model of the target anatomicalobject based on the updated first anatomical parameters.

Another method includes: obtaining an initial model of at least aportion of a patient's coronary vasculature; determining one or morefirst anatomical parameters of the portion of the patient's coronaryvasculature from at least one of one or more first images of the portionof a patient's coronary vasculature; determining one or more secondanatomical parameters of the portion of the patient's coronaryvasculature from at least one of one or more second images of thepatient's coronary vasculature; updating the one or more firstanatomical parameters based at least on the one or more secondanatomical parameters; and generating a final model of the patient'scoronary vasculature based on the updated first anatomical parameters.

In accordance with another embodiment, a system for modeling at least aportion of a patient's anatomy, comprises: a data storage device storinginstructions for modeling based on patient-specific anatomic image data;and a processor configured to execute the instructions to perform amethod including: obtaining an initial model of at least a portion of apatient's coronary vasculature; determining one or more first anatomicalparameters of the portion of the patient's coronary vasculature from atleast one of one or more first images of the portion of a patient'scoronary vasculature; determining one or more second anatomicalparameters of the portion of the patient's coronary vasculature from atleast one of one or more second images of the patient's coronaryvasculature; updating the one or more first anatomical parameters withthe one or more second anatomical parameters; and generating a finalmodel of the patient's coronary vasculature based on the updated firstanatomical parameters.

In accordance with yet another embodiment, a non-transitory computerreadable medium for use on a computer system containingcomputer-executable programming instructions for modeling at least aportion of a patient's anatomy is provided. The method includes:obtaining an initial model of at least a portion of a patient's coronaryvasculature; determining one or more first anatomical parameters of theportion of the patient's coronary vasculature from at least one of oneor more first images of the portion of a patient's coronary vasculature;determining one or more second anatomical parameters of the portion ofthe patient's coronary vasculature from at least one of one or moresecond images of the patient's coronary vasculature; updating the one ormore first anatomical parameters based on the one or more secondanatomical parameters; and generating a final model of the patient'scoronary vasculature based on the updated first anatomical parameters.

Yet another method includes: receiving an initial model of celllocations and diameters of human cells; acquiring at least twohistopathology images of at least a portion of a patient's anatomy;performing localization of cells in each of the at least twohistopathology images to identify cell center locations and diameters ofcells in each image; creating a combined estimate of cell centerlocations and diameters of cells matched between each of the at leasttwo histopathology images; and generating a final cells model of cellcenter locations and diameters based on the combined estimate.

In accordance with another embodiment, a system for modeling at least aportion of a patient's anatomy, comprises: a data storage device storinginstructions for modeling based on patient-specific anatomic image data;and a processor configured to execute the instructions to perform amethod including: receiving an initial model of cell locations anddiameters of human cells; acquiring at least two histopathology imagesof at least a portion of a patient's anatomy; performing localization ofcells in each of the at least two histopathology images to identify cellcenter locations and diameters of cells in each image; creating acombined estimate of cell center locations and diameters of cellsmatched between each of the at least two histopathology images; andgenerating a final cells model of cell center locations and diametersbased on the combined estimate.

In accordance with yet another embodiment, a non-transitory computerreadable medium for use on a computer system containingcomputer-executable programming instructions for modeling at least aportion of a patient's anatomy is provided. The method includes:receiving an initial model of cell locations and diameters of humancells; acquiring at least two histopathology images of at least aportion of a patient's anatomy; performing localization of cells in eachof the at least two histopathology images to identify cell centerlocations and diameters of cells in each image; creating a combinedestimate of cell center locations and diameters of cells matched betweeneach of the at least two histopathology images; and generating a finalcells model of cell center locations and diameters based on the combinedestimate.

Additional objects and advantages of the disclosed embodiments will beset forth in part in the description that follows, and in part will beapparent from the description, or may be learned by practice of thedisclosed embodiments. The objects and advantages of the disclosedembodiments will be realized and attained by means of the elements andcombinations particularly pointed out in the appended claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the disclosed embodiments, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate various exemplary embodiments andtogether with the description, serve to explain the principles of thedisclosed embodiments.

FIG. 1 is a block diagram of an exemplary system and network forgenerating models from multiple patient-specific anatomic images,according to an exemplary embodiment of the present disclosure.

FIG. 2A is a block diagram of an exemplary method for creating apatient-specific model from multiple images, according to an exemplaryembodiment of the present disclosure.

FIG. 2B is a block diagram of an exemplary method for processing variousimages and/or sets of images to produce final object parameters,according to an exemplary embodiment of the present disclosure.

FIGS. 3A-3C are block diagrams of exemplary methods for coronary vesselmodeling, where a final volumetric model includes probabilities thateach voxel belongs to patient coronary vessel lumen, according to anexemplary embodiment of the present disclosure.

FIGS. 4A-4C are block diagrams of exemplary methods for coronary vesselmodeling, where the final model is an object model of the centerlinetree and lumen diameters of the coronary vessel, according to anexemplary embodiment of the present disclosure.

FIG. 5 is a block diagram of an exemplary method for histopathologicalmodeling from multiple optical images, according to an exemplaryembodiment of the present disclosure.

DESCRIPTION OF THE EMBODIMENTS

Reference will now be made in detail to the exemplary embodiments of theinvention, examples of which are illustrated in the accompanyingdrawings. Wherever possible, the same reference numbers will be usedthroughout the drawings to refer to the same or like parts.

As described above, a new generation of noninvasive tests have beendeveloped that is based on modeling anatomy. Thus, a need exists foraccurate modeling. Specifically, there is a need for accurately modelingcoronary anatomy to assess patient anatomy, myocardial perfusion, andcoronary artery flow. Such a method and system may benefit cardiologistswho diagnose and plan treatments for patients with suspected coronaryartery disease.

However, because image-based models generated from single images may beinsufficient, a need exists for a method for modeling anatomy that mayintegrate imaging data from multiple sources. For example, compilingmultiple images to produce a model may enhance complimentary informationin each of the multiple images and reduce imaging artifact in variousimages. Furthermore, images from multiple imaging modalities may becompiled to create more accurate models and models that take advantageof the imaging strengths unique to each imaging technique. For instance,CT and MR images are sometimes considered complimentary. Creating amodel by integrating CT and MR images means that a user no longer has tochoose between the benefits of CT imaging versus the benefits of MRimaging in analyzing anatomical data. As an exemplary embodiment, amethod for modeling coronary anatomy is described in order tononinvasively assess coronary anatomy, myocardial perfusion, andcoronary artery flow. However, such a method and system may be suitablefor any anatomy of interest. By extension, reinforcing the advantages ofeach imaging technique by integrating multiple images may also reducethe impact of disadvantages (e.g., imaging artifacts) associated withvarious imaging techniques.

Thus, the present disclosure is directed to a new approach of usingmultiple images in order to create and provide an accurate anatomicalmodel. The extraction of an anatomical model from an image is a commonproblem in biomedical imaging. Anatomical models may be extracted tomeasure properties of patient anatomy (e.g., tumor or cardiac volume) orto support biophysical simulation (e.g., fluid simulation, biomechanicalsimulation, electrophysiological simulation, etc.). In order toaccurately measure anatomical properties or predict physiologicalphenomena via simulation, a precise, patient-specific model must becreated of the target anatomy. The present disclosure involves the useof multiple images to achieve a patient-specific anatomical model. Forexample, the present disclosure may take advantage of complementaryinformation in each of the multiple images or a reduction in differenttypes of imaging artifact in the different images. Although it ispossible to construct patient-specific models from geometry derived froma single image, use of multiple images produces a hyper-precisepatient-specific anatomical model.

The present disclosure is directed to integrating imaging data frommultiple sources to create a single, precise geometric model.Specifically, the present disclosure may receive various types of imagesor different portions of a target object. The present disclosure mayaverage respective reference images with multiple patient images tocreate a single geometric model.

Referring now to the figures, FIG. 1 depicts a block diagram of anexemplary system and network for predicting coronary plaquevulnerability from patient-specific anatomic image data. Specifically,FIG. 1 depicts a plurality of physicians 102 and third party providers104, any of whom may be connected to an electronic network 100, such asthe Internet, through one or more computers, servers, and/or handheldmobile devices. Physicians 102 and/or third party providers 104 maycreate or otherwise obtain images of one or more patients' cardiacand/or vascular systems. The physicians 102 and/or third party providers104 may also obtain any combination of patient-specific information,such as age, medical history, blood pressure, blood viscosity, etc.Physicians 102 and/or third party providers 104 may transmit thecardiac/vascular images and/or patient-specific information to serversystems 106 over the electronic network 100. Server systems 106 mayinclude storage devices for storing images and data received fromphysicians 102 and/or third party providers 104. Server systems 106 mayalso include processing devices for processing images and data stored inthe storage devices.

FIG. 2A is a block diagram of an exemplary method 200 for creating apatient-specific model from multiple images, according to an exemplaryembodiment of the present disclosure. For example, patient images thatare obtained via various imaging techniques or at different points intime may be compiled to create a final model. The final model createdmay depict different parts of anatomy or various aspects of the anatomy,depending on the input images. Some embodiments of the method mayinclude first obtaining a model of the target anatomy object, to whichpatient images may be mapped. Other embodiments of the method mayfurther include reference images that may serve as a point of comparisonfor such mapping. Method 200 may be performed by server systems 106,based on information, images, and data received from physicians 102and/or third party providers 104 over electronic network 100. The methodof FIG. 2A may include receiving a representation of a target object formodeling (step 202). In one embodiment, the representation may be storedon an electronic storage device (e.g., hard drive, RAM, network drive,etc.). The representation may include an object localization model(e.g., a boundary model or volumetric model). The representation mayalso include, but is not limited to, an appearance model or shape model.In one embodiment, the representation may be determined by a set ofparameters estimated from the images. The object localization model mayinclude a resultant object model based on the estimated parameters. Forinstance, the resultant object model may be comprised of a fullydetermined set of parameters. An exemplary set of parameters todetermine an object model is the assignment of a binary indicator valueto every pixel, the assignment of a probability or level set value toevery pixel. Another set of parameters that may be used to represent anobject model is a set of 3D coordinates and triangles to represent thetriangulated surface of a 3D object.

Step 204 of method 200 may involve receiving a reference image thatdepicts the target object. For example, the reference image may be 2-D,3-D, or 4-D, and the image may be stored in an electronic storagedevice. In one case, the reference image may be directly associated withthe target object. In another case, the reference image may be selectedbased on inferences from the resultant object model.

Then, step 206 may involve receiving a collection of two or more 2-D,3-D, or 4-D images that depict at least part of the target object. Forinstance, this collection of images may be specific to the patient. Inone case, the images are stored and/or transferred via an electronicstorage device. As used herein, the term, “image,” refers to an imageregardless of dimension. In addition, each element making up the imagemay be referred to as a “pixel” or “voxel,” regardless of the image sizeor resolution. For example, each element of a 2-D image may be a pixel,regardless of the image dimensions. Analogously, each element of a 3-Dimage or volumetric model may be regarded as a “voxel,” for images ormodels of any size or resolution. Step 208 of method 200 may includeprocessing the representation from step 202, the reference image fromstep 204, and image collection of step 206 to output final object modelparameters. For example, step 208 may include outputting the parametersto an electronic storage device and/or performing the processing using acomputational device (including but not limited to a computer, laptop,DSP, cloud server, tablet, smart phone, etc.). In one embodiment, method220 of FIG. 2B may be an exemplary method for performing the processingof step 208. In other words, method 200 may employ method 220 to processgathered information and produce the final object parameters output instep 208.

FIG. 2B is a block diagram of an exemplary method 220 for processingvarious images and/or sets of images to produce final object parameters,according to an exemplary embodiment of the present disclosure. Method220 may also be performed by server systems 106, based on information,images, and data received from physicians 102 and/or third partyproviders 104 over electronic network 100. The method of FIG. 2B mayinclude creating an initial set of parameters for the object model usingthe reference image (step 222). For example, in one embodiment, initialparameters may be determined by using an image segmentation technique.For example, step 222 may include defining a set of parameters ofinterest for the object model, and then determining the parameters forthe reference image. Alternately, step 222 may include determining a setof parameters available from the reference image. Step 222 may furtherinclude determining values of each of the parameters. In some cases,this set of parameters may serve as initialized parameters.

Step 224 may include creating an estimate of some of the object modelparameters for each image in the collection (received in step 206). Theinitialized parameters from step 222 may or may not be used to createthe estimate. For example, estimates may be made of more objectparameters than are included in the set of initialized parameters.Alternately, estimates may be made for only a subset of the initializedparameters or the full set of initialized parameters.

In one embodiment, step 226 may include updating or merging theparameters from each image in the collection with parameters andparameter values estimated from the reference image. In one embodiment,image parameters from a first set of image parameters may be updatedbased on image parameters obtained from a second set of images. Forexample, image parameters may be merged by combining and/or averagingcorresponding image parameters obtained from multiple sets of images.Updating parameters may include merging, combining, averagingparameters. Furthermore, updating parameters may include, both changesor verification of existing parameters, as well as generating newparameters. By merging the parameters, step 226 may lead to step 228 ofcreating a combined estimate of the object. In one embodiment, steps222-228 may be repeated until the object model parameters converge. Theobject model parameters may converge into final object parameters. Thesefinal object parameters may serve as the output at step 208 of method200, where final object parameters are output to an electronic storagedevice.

Method 200 may be used to produce various models, depending on thepatient images used in step 206 and the processing of those images inmethod 220. The following disclosure presents several exemplaryembodiments of, or alternatives to method 200. In general, FIGS. 3A-3Care block diagrams of exemplary methods for obtaining volumetric modelsof probabilities that given voxels belong to a patient lumen; FIGS.4A-4C are block diagrams of exemplary methods for producing final objectmodels of patient centerline trees and lumen diameters; and FIG. 5 is ablock diagram of an exemplary method to generate a final model of celllocations and diameters. All of the images and final object modelsdiscussed may be stored in and/or output to electronic storage devices.

FIGS. 3A-3C depict methods for coronary vessel modeling, where a finalvolumetric model includes probabilities that each voxel belongs to apatient coronary vessel lumen. For example, different imaging techniquesmay produce patient images that each portray a coronary vessel lumen insome capacity. Each element (e.g., voxel) of each image may carry someprobability that it is part of a vessel. Forming a composite of theimages may therefore produce an accurate model of a patient coronaryvessel lumen. The model may further include an assessment of the model'saccuracy at any point in the model. In general, FIG. 3A is an exemplarymethod of estimating the probabilities using CTA reconstructions; FIG.3B is an exemplary method of modeling using cardiac CTA images atdifferent points in time; and FIG. 3C models the coronary vessel fromone or more cardiac CTA images and a MR images.

FIG. 3A is a block diagram of an exemplary method 300 for obtainingvolumetric models of probabilities based on CTA reconstructions. In oneembodiment, coronary vessels may be segmented using multiple cCTA imagesthat represent multiple reconstructions. First, a parameterizedrepresentation of a target object may be received and stored on anelectric storage device (e.g., hard drive, RAM, network drive, etc.)(step 301). The target object for this instance may be the coronaryvessel lumen. For example, the representation for step 301 may be avolumetric model of a patient's coronary vessel lumen, where each voxelrepresents the probability that the voxel belongs to the patient'scoronary vessel lumen. Probabilities may or may not be displayed. Forexample, in one embodiment, probabilities may be displayed by showingrespective probabilities of each voxel belonging to the patient'scoronary vessel lumen by displaying a high probability as a high imageintensity and a low probability as a low image intensity.

Step 303 may involve receiving multiple 3-D images from a cardiac CTscanner, where each image represents different reconstructions of thecCTA image for the patient. Reconstructions may include, for example,images with different kernels for filtered backprojection and/oriterative reconstruction methods. Employing multiple reconstructions isuseful in that each image reconstruction technique has differentadvantages and disadvantages for different types of image features. Eachimage reconstruction technique may be better for some types of imagefeatures and worse at others (e.g., blooming, streaking, noise, etc.).Using the best aspects of each reconstruction may help achieve a moreprecise object geometry.

Step 305 may involve determining the probability that each voxel belongsto the patient's coronary vessel lumen. For example, the probability maybe determined by performing a segmentation of the coronary vessel lumenin each image. For example, any known technique for performingsegmentation to obtain voxel probabilities may be used, such as randomwalker algorithms or machine learning algorithms that map voxelintensities and their neighbors to probabilities. The segmentation maybe performed using any existing technique, and the segmentation may beperformed for the coronary vessel lumen independently with respect toeach image. For example, segmentation may be performed automatically bya computer system either based on user inputs or without user inputs.For instance, in an exemplary embodiment, the user may provide inputs tothe computer system in order to generate a first initial model. Foranother example, the computer system may display to the user a 3-D imageor slices thereof produced from the CCTA data. The 3-D image may includeportions of varying intensity of lightness. For example, lighter areasmay indicate the lumens of the aorta, the main coronary arteries, and/orthe branches. Darker areas may indicate the myocardium and other tissueof the patient's heart.

Step 307 may involve averaging the probabilities (from step 305) acrossthe images to create a combined estimate of the volumetric model of theprobability of each voxel belonging to the patient lumen. In oneembodiment, steps 305 and 307 may be performed using a computer.Finally, the final, averaged volumetric model of the probabilities maybe output (step 309), for example, to an electronic storage device. Forexample, the averaged volumetric model of the probabilities may beoutput in the form of a color overlay showing the boundary of a levelset on the probabilities, or as a set of raw probabilities.

FIG. 3B is a block diagram of an exemplary method 320 for obtainingvolumetric models of probabilities based on CTA images obtained atdifferent points in time. As in method 320, step 321 involves obtaininga parameterized representation of a target object (e.g., a coronaryvessel lumen). The representation may be a volumetric model of apatient's coronary vessel lumen in which each voxel represents theprobability that the voxel belongs to the patient's coronary vessellumen. Similar to step 303, step 323 may involve receiving multiple 3-Dimages from a cardiac CT scanner where each image represents a differentreconstruction of the cCTA image for a patient. Unique to method 320,the images may represent acquisitions from a single patient, at multiplepoints in time (e.g., time points within the cardiac cycle or an initialacquisition and follow-up scan). Using images at multiple time pointsmeans that each image may contain independent information that maycontain less artifact or better quality in different regions of theimage. Method 320 may include using the best aspects of eachreconstruction to achieve a final model of precise object geometry. Theimages for step 323 may be stored in an electronic storage device. Forexample, a storage device may determine a new image acquisition andupdate a final volumetric model by taking into account the newacquisition.

Step 325 a may reflect step 305 in determining the probability that eachvoxel belongs to the patient's coronary vessel lumen. For example, step325 a may include finding the probability by performing a segmentationof the coronary vessel lumen independently in each image (using anyexisting technique). Next, step 325 b may involve choosing a referenceimage. For example, the reference image may be arbitrary among theacquired images, retrieved from a reference image repository, selectedintentionally from a set of acquired images, etc. Then, 3-D registrationmay be used to register each image to the reference image (step 325 c).In certain embodiments, steps 325 a-325 c may be analogous to steps 345a-345 c of FIG. 3C.

Step 327, like step 307, may involve creating a combined estimate of thevolumetric model by averaging probabilities across images. However, theimages for step 327 may include patient images and the reference image.The image registration may be used to map each voxel to another voxel,meaning mapping a location in each image to a location (or locations) inthe reference image and/or other images. The mapping may be performedusing any method such that voxels in two images may be identified asbeing representations of the same part of the target object (e.g.,coronary vessel lumen). Since the voxels correspond to the same part,the voxel probabilities may be combined. Therefore, averaging theprobability of each voxel belonging to the patient lumen may create amerged, averaged voxel probability for each voxel. Finally, step 329 mayinclude outputting the final volumetric model of the probabilities to anelectronic storage device.

FIG. 3C is a block diagram of an exemplary method 340 for obtainingvolumetric models of probabilities based on a plurality of cCTA imagesthat represent the coronary tree and a plurality of cardiac magneticresonance (MR) images that also depict the coronary tree. Step 341involves receiving a parameterized representation of a coronary vessellumen stored on an electronic storage device, similar to steps 301 and321. Again, the representation may be a volumetric model of thepatient's coronary vessel lumen, in which each voxel represents theprobability that the voxel belongs to the patient's coronary vessellumen. At step 343, in addition to receiving one or more 3-D images froma cardiac CT scan, step 343 may further include receiving one or more3-D images from a cardiac MR scan of a patient. Both CT and MR cardiacimages may be acquired to obtain different cardiac information availablefrom each modality (e.g., fine detail with CT and viability with MR).Further, MR imaging may exhibit fewer blooming artifacts near calciumthan CT images exhibit, so MR images may be more useful in some casesfor examining the geometry of calcified lesions. Both the CT scan(s) andMR scan(s) may be stored in an electronic storage device.

As previously stated, steps 345 a-345 c may be similar to steps 325a-325 c, as applied to a context involving MR images. These steps mayall be performed using a computer. Step 345 a, like steps 305 and 325 a,may involve segmentation that determines probability associated witheach voxel that the voxel belongs to the patient's coronary vessellumen. Then, step 345 b may involve choosing an arbitrary image or animage with the greatest spatial resolution as a reference image. Thus,the reference image may, in some cases, have the smallest voxel size.Step 345 c may involve using 3-D image registration to register eachimage to the reference image from step 345 b. For example, step 345 cmay include using 3-D image registration to register each voxelprobability in each image to each respective voxel probability in thereference image. Steps 345 c and 347 may involve creating a combinedestimate of a volumetric model by using the image registration to mapeach voxel probability to a corresponding voxel probability of thereference image. Again, in one embodiment, the mapping may create amerged (e.g., averaged) voxel probability. Step 349, analogous to steps309 and 329, may involve outputting the averaged, final volumetric modelof the probabilities to an electronic storage device.

FIGS. 4A-4C depict methods for coronary vessel modeling, where the finalmodel may be an object model of the centerline tree and lumen diametersof the coronary vessel. FIG. 4A represents vessel modeling from acardiac CTA image and an intravascular ultrasound (IVUS) and/orintravascular optical coherence tomography (OCT) image, while FIGS. 4Band 4C are two embodiments of a method of coronary vessel modeling froma cardiac CTA image and an angiography image.

FIG. 4A is a block diagram of an exemplary method 400 for coronaryvessel modeling from a cardiac CTA image and an IVUS/OCT image. Forexample, a parameterized representation of the target object (e.g., acoronary vessel lumen) may be obtained (step 401). The representationmay be a model of a patient's coronary vessel lumen in which each vesselis represented by a centerline and the lumen boundary is represented bya lumen diameter associated with each centerline location.

Step 403 may involve receiving one or more 3-D images from a cardiac CTscan and one or more intravascular images of the same patient. Forexample, the coronary vessels may be segmented using one or more cCTAimages that represent the entire coronary tree and one or more IVUS orOCT images of at least a portion of the coronary tree. Going forward,the term, “intravascular images,” may be taken to refer to the IVUSand/or OCT images. Cardiac CTA and intravascular cardiac images may beacquired due to different cardiac information generated by each modality(e.g., complete 3-D vessel tree from CT and high-resolution vesselgeometry from intravascular imagery). Furthermore, intravascular imagingmay exhibit fewer blooming artifacts near calcium than are exhibited byCT. As a result, intravascular imaging may be especially useful for thepurpose of examining the geometry of calcified lesions.

For step 405 a, a segmentation of the coronary vessel lumen may beperformed independently in each image to create either a completecenterline tree (e.g., for the coronary CT images) or a part of thecenterline tree (e.g., for intravascular images). This segmentation maydetermine the diameter of the coronary vessel lumen at each location ofthe centerline. Step 405 b may involve choosing an arbitrary cCTA imageas the reference image, and step 405 c may involve using registration toregister each object model or part of an object model to the objectmodel obtained from the reference image. Again, each object model may becomprised of one or more centerlines and/or one or more lumen diameters.For step 407, a combined estimate of the object model may be created byaveraging the lumen diameter at each centerline location with anestimate from each source (e.g., as determined from step 405 a). All thesteps of method 400 may be performed using a computer, especially steps405 a-407. Lastly, a final, averaged object model of the centerline treeand lumen diameters may be output (step 409).

FIG. 4B is a block diagram of an exemplary method 420 for coronaryvessel modeling from a cardiac CTA image and an angiography image. Inone embodiment, step 421 may include receiving a parameterizedrepresentation of a coronary vessel lumen as a target object, where therepresentation may be a model of the lumen. For example, each vessel maybe represented by a centerline and the lumen boundary may be representedby a lumen diameter associated with each centerline location. Step 423may involve receiving one or more 3-D images from a cardiac CT scan andone or more angiography images of the same patient. Again, coronaryvessels may be segmented using one or more cCTA images that representthe entire coronary tree, but in contrast to method 400, at least aportion of the coronary tree may be segmented using one or more 2-Dangiography images, rather than (or in addition to) IVUS and/or OCTimages. Cardiac CTA and angiography images may be acquired due todifferent cardiac information generated by each modality (e.g., complete3-D vessel tree from CT and high-resolution vessel geometry from theangiography). Furthermore, angiography imaging may exhibit fewerblooming artifacts near calcium than exhibited by CT, making it usefulfor examining the geometry of calcified lesions.

In certain embodiments, steps 425 a-425 c may be analogous to steps 445a-445 c of FIG. 4C; and steps 425 a-427 and steps 445 a-447 may all beperformed using a computer. Step 425 a may involve performingsegmentation of the coronary vessel lumen independently in each image tocreate a complete centerline tree (e.g., for the coronary CT images) ora part of the centerline tree (e.g., for the angiography images). Thesegmentation may determine the diameter of the coronary vessel lumen ateach point of the centerline. Then, an arbitrary cCTA image may bechosen as a reference image (step 425 b) and registration may be used toregister each object model or part of an object model to the objectmodel obtained from the reference image. Each object model may becomprised of one or more centerline(s) and one or more lumendiameter(s). If viewing angles are available for the angiography images,the information from that analysis may be taken into account whenregistering the models (step 425 c). Step 427 may involve creating acombined estimate of the object model by averaging lumen diameter(s) ateach centerline location with an estimate of the diameter from eachsource (e.g., as given by step 425 a). Step 429 may include outputting afinal, averaged object model of centerline tree and lumen diameters.

FIG. 4C is a block diagram of an exemplary method 440, also for coronaryvessel modeling from a cardiac CTA image and an angiography image.Method 440 may be used as an alternative to method 420. Like method 420,method 440 may begin with obtaining a parameterized representation thatis a model of a patient's coronary vessel lumen, in which each lumen isrepresented by a centerline, and the lumen boundary may be representedby a lumen diameter associated with each centerline location (step 441).Step 443 a may involve receiving a 3-D image from a cardiac CT scan andone or more angiography images of the same patient with viewing anglesfor each angiography image. A further step 443 b may involve receivingadditional patient-specific information, such as, for example, an upperand/or lower bound on total vascular volume. Step 445 a, like step 425a, may include performing a segmentation of the coronary vessel lumen tocreate a complete centerline tree (e.g., for the coronary CT images) orpart of the centerline tree (e.g., for the angiography images), wherethe segmentation may determine the diameter of the coronary vessel lumenat each location of the centerline. In contrast to the arbitrarilydesignated reference images from step 425 b, step 445 b may choose thecCTA image as the reference image. Then, step 445 c may involveevaluating whether all the 2-D projections of the geometric model ontothe 2-D angiography spaces match respective 2-D segmentations andconstraints of the additional patient-specific information. In oneembodiment, the matching may be done in a Bayesian framework if theprojections are described as probability density functions. If a matchis achieved, a geometric representation may be extracted from the cCTAimage (step 447). Step 449 may include outputting a final object modelof the centerline tree and lumen diameters.

FIG. 5 is a block diagram of an exemplary method 500 for modelinganatomy from multiple optical (e.g., histopathological) images. Forexample, optical images may be obtained for various types of tissue.Tissue samples may be from surgery, biopsy, or autopsy.Histopathological examination of tissue samples may include processingmultiple optical images of the tissue in order to create a final modelof cells of the tissue sample. Such information may offer insight intodiseases affecting the tissue. The method may begin with obtaining aparameterized representation of target objects (step 501). In oneembodiment, the target objects may be a set of cell locations in atissue sample. For example, step 501 may include retrieving a model ofcell locations and a diameter for each location. Step 503 may involvereceiving two or more 2-D or 3-D histopathology images of the sametissue sample. Cell locations of a tissue sample may be computed frommultiple 2-D histopathology optical images. Multiple images might beacquired to provide a hyper-precise localization of all cells in thetissue sample. Step 505 a may then include performing a localization ofall cells in an image to create a center location and diameter estimatefor each cell. Then, an arbitrary histopathology image may be chosen asthe reference image (step 505 b), and registration may be used toregister each cells model to the cells model obtained from the referenceimage (step 505 c). In one case, each cells model may comprise a centerlocation and/or diameter for each cell in a field of view associatedwith the tissue sample. Then, step 507 may include averaging centerlocations and diameter estimates for each cell from all cell models.Based on the averaging, step 507 may further involve creating a combinedestimate of the cells model. Again, an output of the final, averagedmodel of cell locations and/or diameters may be made (step 509).

Thus, modeling may be improved using multiple images, whether thevariations are images acquired from reconstructions, various imagingmodalities, or acquisitions at different points in time. Creating modelsby aggregating multiple images permits the combination of benefits ofdifferent imaging modalities models, reduces errors or imperfections inindividual image acquisitions, and strengthens the credibility of thefinal model. The technique of modeling using multiple images may beapplied to model any target object, anatomy of interest, or informationassociated with that target object and/or anatomy or interest.

Other embodiments of the invention will be apparent to those skilled inthe art from consideration of the specification and practice of theinvention disclosed herein. It is intended that the specification andexamples be considered as exemplary only, with a true scope and spiritof the invention being indicated by the following claims.

What is claimed is:
 1. A computer-implemented method of modeling atleast a portion of a patient's vasculature, using a computer system, themethod comprising: determining one or more lumen diameters of thepatient's coronary vasculature from one or more three-dimensional,computed tomography images of the patient's coronary vasculature;determining one or more anatomical parameters of the patient's coronaryvasculature from one or more intravascular ultrasound (“IVUS”),angiography or optical coherence tomography (“OCT”) images of thepatient's coronary vasculature obtained from within a lumen of thepatient's coronary vasculature; extracting a geometric model of thepatient's coronary vasculature from one or more of thethree-dimensional, computed tomography images or IVUS, angiography, orOCT images designated as a reference image; registering an object modelto the geometric model based on the one or more determined lumendiameters or one or more determined anatomical parameters; using theregistration of the object model to the geometric model to update theone or more lumen diameters derived from three three-dimensional,computed tomography images based on at least one of the one or moredetermined anatomical parameters derived from the IVUS, angiography, orOCT images; and generating a model of the patient's coronary vasculaturebased on at least one of the one or more updated lumen diameters.
 2. Themethod of claim 1, wherein the one or more anatomical parameters of thepatient's coronary vasculature are determined from one or moreintravascular ultrasound images.
 3. The method of claim 1, wherein theone or more anatomical parameters of the patient's coronary vasculatureare determined from one or more angiography images.
 4. The method ofclaim 1, wherein the model of the patient's coronary vasculatureincludes at least a centerline and representation of a lumen boundarycorresponding to a location on the centerline.
 5. The method of claim 1,wherein determining the one or more lumen diameters or the one or moreanatomical parameters includes performing a segmentation of the one ormore three-dimensional, computed tomography images or the one or moreIVUS, angiography, or OCT images of the patient's coronary vasculature,to generate a whole or partial vessel centerline tree.
 6. The method ofclaim 1, further comprising: designating one of the one or morethree-dimensional, computed tomography images or one of the one or moreIVUS, angiography, or OCT images as the reference image; generating theobject model based on one of the one or more lumen diameters or one ofthe one or more anatomical parameters; registering the object model toan object model associated with the reference image; and averaging oneor more lumen diameters of the object model with one or more lumendiameters of the object model associated with the reference image. 7.The method of claim 1, wherein the model includes a representation ofthe lumen of the patient's coronary vasculature.
 8. The method of claim1, further including: designating one of the one or more firstthree-dimensional, computed tomography images as the reference image;extracting a geometric representation from the reference image; andgenerating the model of the patient's coronary vasculature based on thegeometric representation.
 9. The method of claim 1, further including:receiving patient-specific information, including upper and/or lowerbound information on total vascular volume; and generating the model ofthe patient's coronary vasculature further based on the receivedpatient-specific information.
 10. A system of modeling at least aportion of a patient's anatomy, using a computer system, the systemcomprising: a data storage device storing instructions for modelingbased on patient-specific anatomic image data; and a processorconfigured to execute the instructions to perform a method including:determining one or more lumen diameters of the patient's coronaryvasculature from one or more three-dimensional, computed tomographyimages of the patient's coronary vasculature; determining one or moreanatomical parameters of the patient's coronary vasculature from one ormore intravascular ultrasound (“IVUS”), angiography, or opticalcoherence tomography (“OCT”) images of the patient's coronaryvasculature obtained from within a lumen of the patient's coronaryvasculature; extracting a geometric model of the patient's coronaryvasculature from one or more of the three-dimensional, computedtomography images or IVUS, angiography, or OCT images designated as areference image; registering an object model to the geometric modelbased on the one or more determined lumen diameters or one or moredetermined anatomical parameters; using the registration of the objectmodel to the geometric model to update the one or more lumen diametersderived from three three-dimensional, computed tomography images basedon at least one of the one or more determined anatomical parametersderived from the IVUS, angiography, or OCT images; and generating amodel of the patient's coronary vasculature based on at least one of theone or more updated lumen diameters.
 11. The system of claim 10, whereinthe one or more anatomical parameters of the patient's coronaryvasculature are determined from one or more intravascular ultrasoundimages.
 12. The system of claim 10, wherein the one or more anatomicalparameters of the patient's coronary vasculature are determined from oneor more angiography images.
 13. The system of claim 10, wherein themodel of the patient's coronary vasculature includes at least acenterline and representation of a lumen boundary corresponding to alocation on the centerline.
 14. The system of claim 10, whereindetermining the one or more lumen diameters or the one or moreanatomical parameters includes performing a segmentation of the one ormore three-dimensional, computed tomography images or the one or moreIVUS, angiography, or OCT images of the patient's coronary vasculature,to generate a whole or partial vessel centerline tree.
 15. The system ofclaim 10, wherein the processor is further configured for: designatingone of the one or more first three-dimensional, computed tomographyimages or one of the one or more IVUS, angiography, or OCT images as thereference image; generating the object model based on one of the one ormore lumen diameters or one of the one or more anatomical parameters;registering the object model to an object model associated with thereference image; and averaging one or more lumen diameters of the objectmodel with one or more lumen diameters of the object model associatedwith the reference image.
 16. The system of claim 10, wherein the modelincludes a representation of the lumen of the patient's coronaryvasculature.
 17. The system of claim 10, the processor is furtherconfigured for: designating one of the one or more three-dimensional,computed tomography images as the reference image; extracting ageometric representation from the reference image; and generating themodel of the patient's coronary vasculature based on the geometricrepresentation.
 18. The system of claim 10, the processor is furtherconfigured for: receiving patient-specific information, including upperand/or lower bound information on total vascular volume; and generatingthe model of the patient's coronary vasculature further based on thereceived patient-specific information.
 19. A non-transitory computerreadable medium for use on a computer system containingcomputer-executable programming instructions for performing a method ofmodeling at least a portion of a patient's anatomy, the methodcomprising: a data storage device storing instructions for modelingbased on patient-specific anatomic image data; and a processorconfigured to execute the instructions to perform a method including:determining one or more lumen diameters of the patient's coronaryvasculature from one or more three-dimensional, computed tomographyimages of the patient's coronary vasculature; determining one or moreanatomical parameters of the patient's coronary vasculature from one ormore intravascular ultrasound (“IVUS”), angiography, or opticalcoherence tomography (“OCT”) images of the patient's coronaryvasculature obtained from within a lumen of the patient's coronaryvasculature; extracting a geometric model of the patient's coronaryvasculature from one or more of the three-dimensional, computedtomography images or IVUS, angiography, or OCT images designated as areference image; registering an object model to the geometric modelbased on the one or more determined lumen diameters or one or moredetermined anatomical parameters; using the registration of the objectmodel to the geometric model to update the one or more lumen diametersderived from three three-dimensional, computed tomography images basedon at least one of the one or more determined anatomical parametersderived from the IVUS, angiography, or OCT images; and generating amodel of the patient's coronary vasculature based on at least one of theone or more updated lumen diameters.
 20. The non-transitory computerreadable medium of claim 19, wherein the final model includes arepresentation of the lumen of the patient's coronary vasculature.